Non-autoregressive Translation with Layer-Wise Prediction and Deep Supervision

How do we perform efficient inference while retaining high translation quality? Existing neural machine models, such as Transformer, achieve performance, but they decode words one by one, which is inefficient. Recent non-autoregressive models speed up the inference, their quality still inferior. In this work, propose DSLP, a highly and high-performance model for translation. The key insight to train Transformer with Deep Supervision feed additional Layer-wise Predictions. We conducted extensive experiments on four tasks (both directions of WMT'14 EN-DE WMT'16 EN-RO). Results show that our approach consistently improves BLEU scores compared respective base models. Specifically, best variant outperforms autoregressive three tasks, being 14.8 times more in inference.